Measuring populism using nosiy human codings of textual data

Crowd-sourced codings of populism in textual data

Hua et al. have recruited crowd workers on the platform CrowdFlower to code social media posts created by a selected number of accounts of Wesrtern European parties and their leaders according to the following coding scheme:

1. Filter questions:
   1. This post has no text or its content is impossible to understand.
   2. I understand the message of this social media post.
2. Anti-elitism: Does this tweet/post criticize or mention in a negative way the elites?
3. People-centrism: Does this tweet/post mention in a positive way or even praise the people (citizens of the country, the working class, the native …) or the nation?
4. Exclusionism: Does this tweet/post criticize minorities or specific groups of people (muslims, jews, LGBT people, poor people …)?

Coders were asked to answer Yes or No to all questions; all judgments obtained are thus binary. If a coder answered question 1.1 affirmatively for a post, she was asked to skip the given post and proceed with judging the next one. If a coder answered question 1.2 affirmatively for a post, she was asked to proceed with answering questions 2–4 and then proceed with judging the next post.

Of particular interst are the judgments Hua et al. collected in an effort to assertain the reliability of their crowd-source measurement instrument. Interested in computing inter-coder agreement metrics, Hua et al. crowd-sourced judgments from 1 different coders for a set of 1 different social media posts (items). As the following table shows, each item was coded between 1 and four times.

Keeping all judgments that passed the first two filter questions,¹ I was able to retain a total 1489 items from the original validation data.

The Bayesian Beta-Binomial by Annotator model

I assume that anti-elitism, people-centrism, and exclusionism—the three dimensions of Hua et al.’s measurement instrument—are latent binary features of political posts, and hence crowd coders act as human content analysts whose judgments I want to aggregate at the item-level to estimate whether a given item belongs to either of these three categories.

For each dimension, the setup can be described as a four-tuple \(\langle\mathcal{I}, \mathcal{J}, \mathcal{K}, \mathcal{Y}\rangle\), where

• \(\mathcal{I}\) is the set of \(i \in 1,\ \ldots ,\ n\) distributed for crowd coding,
• \(\mathcal{J}\) is the set of \(j \in 1,\ \ldots ,\ m\),
• \(\mathcal{K}\) is the set of \(k \in \{0, 1\}\) defined by the categorical coding scheme used during crowd-coding, and
• \(\mathcal{Y}\) is the set of (or codings) \(y_{i,j} \in \{0, 1\}\) recorded for item \(i\) by coder \(j\).

In total, we have \(d = n \times m\) judgments. In the validation data, it is provided that \(\mathcal{Y}\) contains at least one judgment per item, that is, \(|\mathcal{Y}_i| \geq 1\ \forall\ i \in \mathcal{I}\), and that \(|\mathcal{Y}_i| \geq 2\ \forall\ i \in \mathcal{I}' \subset \mathcal{I}\). Moreover, judgments for each item in \(\mathcal{I}'\) are generated by different coders (i.e., we have repeated annotation at the statement level, but not at the judgment-item level).

Importantly, while coders’ judgments of items are observed, items’ true classes \(c_i \in \mathcal{K}\) are unknown a priori for all \(i = 1,\ \ldots,\ n\). In this setup, a classification of items into classes obtained from a set of judgments \(\rho(\mathcal{Y}) \Rightarrow \mathcal{C}\) is called a ground truth labeling. I obtain ground-truth class estimates by fitting the following model to the judgment data:

\[ \begin{align*} c_i &\sim\ \mbox{Bernoulli}(\pi)\\ \theta_{0j} &\sim\ \mbox{Beta}(\alpha_0 , \beta_0)\\ \theta_{1j} &\sim\ \mbox{Beta}(\alpha_1 , \beta_1)\\ y_{ij} &\sim\ \mbox{Bernoulli}(c_i\theta_{1j} + (1 - c_i)(1 - \theta_{0j}))\\ {}&{}\\ \pi &\sim\ \mbox{Beta}(1,1)\\ \alpha_0/(\alpha_0 + \beta_0) &\sim\ \mbox{Beta}(1,1)\\ \alpha_0+\beta_0 &\sim\ \mbox{Pareto}(1.5)\\ \alpha_1/(\alpha_1 + \beta_1) &\sim\ \mbox{Beta}(1,1)\\ \alpha_1+\beta_1 &\sim\ \mbox{Pareto}(1.5) \end{align*} \] where

• \(c_i\) is the ‘true’ (unobserved) class of statement \(i\),
• \(\pi\) is the ‘true’ prevalence of the positive class,
• \(\theta_{0,j}\) is coder \(j\)’s specificity (true-negative rate), and
• \(\theta_{1,j}\) is her sensitivity (true-positive rate).

Carpenter (2008) refers to this model as the Beta-Binomial by Annotator (BBA) model. This name is due to its property that, given a conjugate beta prior, the posterior densities of items’ class membership follow a beta-binomial distribution. In Carpenter’s original formulation, all priors are choosen to be uninformative, as we often have no domain-specific prior knowledge about items classes, coders’ abilities and positive class prevalence. Note that the distribution of coder specificities is parametrezized in terms of the mean and scale of the Beta-distribution. Specifically, I choose a uniform prior for the mean \(\alpha/(\alpha+\beta)\) and a uniform prior for the inverse square scale \(1/(\alpha+\beta)^2\). As Carpenter explicates, the prior on the means is conveniently expressed using a Beta distribution with \(a = b = 1\), whereas the flat prior on the inverse square scale is expressed as a Pareto prior with \(\alpha = 1.5, c=1\).

Let’s have a look at how the mean and scale parameters are distributed

These priors can also be inspected with regard to what priors they induce on \(\alpha\) and \(\beta\), respectively. To do so, we sample 1000 pairs from these distributions and compute \(\alpha^{sim}\) and \(\beta^{sim}\) values from each pair.

Finally, I use the sampled \(\alpha^{sim}\) and \(\beta^{sim}\) values to obtain draws from a correspondingly parameterized Beta distributions. This yields the following empirical distribution:

Essentially, the prior is pushing coders’ ability parameters to the extremes.

Estimation

Before turning to the analysis of coder abilities in people-centrism, anti-elitism, and exclusionism classification, respectively, I fit the models and report convergence metrics.

Evaluation

Agreement between model- and majority-voting based class assginments

While we have reason to doubt the quality of posterior estimates of the BBA model fitted to crowd-sourced exclusionism classification, we can interprete the posterior estimates of coders’ abilities obtained for their classification of posts into people-centrist and anti-elitist instances to indicate that they performed overwhelmingly well in detecting true-negative instances, less well, though, in detecting true-positive instances. Given this evidence of positivity bias in coders’ classification abilities (they tend to err when classifying items as positive, and are often right when classifying items as negative), I expect that aggregating judgments by determining the majority-winner label induces a positive bias at the aggregate level, since majority voting does not account for coders’ fallibility.

Indeed there are in total only 41 out of 1489 items (i.e., 4.1%) for which model-based and majority-voting classifications disagree. The vast share of this disagreement results from items that are classified as featuring people-centrism with majortiy voting but not when using BBA model-based aggregation (40). Hence, there is evidence of positive bias in majority-voting based classifications. Importantly, this disagreement occurs most often where only one coder judged an item. As a consequence of these differences, the empirical prevalence (not to confuse with \(\pi\)) differs somewhat between classification methods: 0.158 in case of majority voting vs. 0.119 in model-based classification.

Though the the same line of reasoning could be applied, we find no evidence of positive bias in majority-winner classifications of items into the anti-elitism class:

Indeed there are in total only 30 out of 1489 items for which model-based and majority-voting classifications disagree.